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The Center on Education Policy (CEP) report, Has Progress Been Made in Raising 
Achievement for English Language Learners?, finds that some states have seen increases 
in the number of English language learners (ELLs) meeting proficiency standards under 
No Child Left Behind (NCLB), while others have seen decreases. The report notes some 
limitations in the data it uses. The CEP report, however, has some specific weaknesses in 
its research methods that undermine its findings. The CEP report seriously underesti- 
mates the significance of language of instruction as a source of error in ELL achievement 
test scores. Further, it errs in implying that its findings justify an inference of a causal re- 
lationship between observed changes in percentages of ELLs meeting achievement 
benchmarks and improvements in academic achievement for ELLs. Given the limitations 
in the data, it is inappropriate to draw conclusions from the data summarized in the CEP 
report. 



Summary of Review 




Review 



I. Introduction 

Has Progress Been Made in Raising Achieve- 
ment for English Language Learners ?} a re- 
port published by the Center on Education Pol- 
icy (CEP), focuses on the achievement of Eng- 
lish language learners (ELLs) under current 
federal NCLB policy. The report examines 
changes in the percentages of ELLs identified 
as meeting the various levels of proficiency in 
data reported by states from 2006 to 2008. 

II. Findings and Conclusions 
of the Report 

The report reaches eight specific conclu- 
sions (pp. 1,2). Two conclusions focus on 
data quality: 

• “Because of language barriers, test re- 
sults for English language learners may 
not accurately reflect what these students 
know and can do. ... 

• “The number of ELL test-takers has 
changed rapidly in many states, which 
complicates efforts to track achievement 
trends for this group. . . . 

Six conclusions focus on ELL achievement: 

• “In grade 4 reading and math, rising per- 

centages of English language learners 
have reached three achievement levels — 
basic, proficient, and advanced — 

although gains are less prevalent at the 
advanced level in reading. . . . 

• “English language learners have made 

progress in reaching state proficiency 
benchmarks in reading and math at all 
three grade levels analyzed — 

elementary, middle, and high school — 
although gains are less prevalent in high 
school than at the other grade levels. . . . 
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• “Trends for English language learners 
are mostly positive in states with the 
largest number of ELL test-takers. In the 
10 states with the most ELL test -takers, 
positive trends outnumbered negative 
trends. ... 

• “English language learners in grade 4 are 
doing better in math than in reading at 
all three achievement levels. . . . 

• “Very large differences in percentages 
proficient exist between English lan- 
guage learners and other students. . . . 

• “Percentages proficient for English lan- 
guage learners vary widely by state, 
more so than for students who are not 
ELLs. ...” 

III. The Report’s Rationale for 
Its Findings and Conclusions 

The report’s authors indicate that their “main 
unit of analysis is the trend line, which is a 
record of change from 2006 through 2008 in 
the performance of the ELL subgroup ...” for 
all the states in the aggregate. This unit of anal- 
ysis, they believe, will help them with their 
primary task, namely, “to determine whether 
ELLs have made progress across the achieve- 
ment spectrum” (p. 4). While they present 
some cautions, the authors believe that the data 
support the general conclusion that states have 
made progress in the raising achievement of 
ELLs. The authors present descriptive summa- 
ries of student scores. They do not use any sta- 
tistical tests to compare student outcomes. 

IV. The Report’s Use of 
Research Literature 

The report makes limited use of relevant re- 
search literature. Some attention is given to 
research related to special accommodations 
for ELL test-takers and to validity concerns 
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about ELL test scores. The report does not, 
however, take advantage of methodological 
insights of related studies published else- 
where. 2 

V. Review of the Report’s 
Methods 

There are significant concerns related to the 
quality and suitability of the data that the re- 
port uses to support its conclusions. The re- 
port’s authors note some of these limitations, 
but underestimate their significance, and they 
advance conclusions based on the data even 
though the limitations they note should pre- 
vent such conclusions from being offered. 

Language of the Test as a Source of Error 

The report notes that ELLs’ test scores in Eng- 
lish may “not accurately reflect what these 
students know” and notes that “questions re- 
main about the reliability of test scores for 
ELLs and the validity of inference drawn from 
these scores” (p. 1). However, this assessment 
severely understates the problem associated 
with using ELLs’ scores in English; more ac- 
curately, one might characterize the scores as 
misleading and inappropriate for students with 
limited English proficiency. 

Language proficiency in general has been 
shown to influence performance on 
achievement tests/ 1 Pilkington, Piersel, and 
Ponterotto (1988) reported that the home 
language of a child influenced the predictive 
validity of kindergarten achievement meas- 
ures. 4 These studies suggest language profi- 
ciency plays a role in young children’s per- 
formance on achievement tests. The Stan- 
dards for Educational and Psychological 
Testing warns that when testing a non-native 
speaker in English, the test results cannot be 
expected to reflect accurately the abilities 
and competencies being measured if test 
performance depends on the test takers’ 
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knowledge of English. 5 Because states’ aca- 
demic proficiency tests are designed to 
measure knowledge of content standards and 
are specifically not designed in relation to a 
theory of language proficiency, limited Eng- 
lish proficiency must be regarded as a major 
source of error in the measurement of aca- 
demic achievement. 

To illustrate the problem associated with 
relying on assessment data from ELLs tested 
in English, imagine a test of academic 
achievement administered to a group of 
English-speaking elementary school stu- 
dents in Spanish. Because relatively few of 
the students know Spanish, they would have 
considerable difficulty understanding the 
questions. We would expect their limited 
Spanish ability to negatively affect their test 
scores. Furthermore, because the test claims 
to measure academic achievement and not 
Spanish, we would regard these circums- 
tances as a source of measurement error. 
The scores would be of little or no value, as 
they would not reflect students’ subject mat- 
ter knowledge. 

We know with some certainty that all the 
students in the ELL subgroup have been as- 
sessed as having limited English proficiency 
by a state language proficiency test. We can 
therefore conclude that the achievement test 
scores summaries in the CEP report are not 
valid, as the students have only limited abili- 
ty to understand the language of the academ- 
ic achievement test. 

Changing Student Cohort Composition 
Makes Comparisons Inappropriate 

Although the reports’ authors are clearly 
aware that the data relate to changing percen- 
tages of ELLs and to states’ relative success in 
increasing the percentages of ELLs meeting 
proficiency benchmarks, they sometimes state 
conclusions in terms of student-level growth. 
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For example, the report asserts that the data 
show that “English language learners have 
made progress in reaching state proficiency 
benchmarks in reading and math at all three 
grade levels ...” (p. 2). However, the data do 
not pennit conclusions regarding year-to-year 
changes in ELL performance because student 
cohorts are not followed over time in the 
study. The composition of the ELL subgroup 
changes dramatically from year to year be- 
cause students are exited from the subgroup 
each year based on achievement score bench- 
marks (“skimming effect”) 6 ; because ELLs 
and other lower income students are highly 
mobile, moving across districts, states, and 
national boundaries; and because state policy 
fluctuates with regard to conditions under 
which ELL scores are included in data sum- 
maries. These facts are also suggested by the 
report’s observation that numbers of test- 
takers changed dramatically over time in some 
states. Given these considerations, it is not on- 
ly unlikely that the ELL subgroup is made up 
of the same students each year, but it is vir- 
tually certain that it is substantially different 
each year. Hence, drawing growth-related 
conclusions from data with these dramatic li- 
mitations is inappropriate. 

Causal Inference is Not Valid 

The report does not focus on changes in data 
reflecting student achievement outcomes 
(test score data, for instance), but rather on 
the percentage of students meeting state cri- 
teria each year. The question asked in its 
title — “Has progress been made in raising 
achievement for English language learners?” 

— is not actually addressed in the report and 
cannot be addressed using these data or re- 
search methods. The question asks us to in- 
fer a direct causal relationship between 
changes in reported percentages of ELLs 
meeting year benchmarks and improvements 
in academic achievement for ELLs. Howev- 
er, no methodological approach to causal 
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inference is followed in the report, and mul- 
tiple competing explanations for the annual 
changes, each as plausible as the next, are 
not addressed. Consider, for instance, the 
following possible alternative explanations 
for the change in students’ test scores: 

• Standardized test scores tend to increase 
over time regardless of interventions. 7 

• Schools have focused more strategically 
on the sample of items likely to be ad- 
dressed in the assessments. 

• Districts are postponing reclassification 
of high-scoring ELLs. 

• Lower scoring ELLs are relocating to 
states reporting declines in the percen- 
tage of students meeting proficiency. 

• States with higher percentages of ELLs 
meeting proficiency have easier tests. 

• States with higher percentages of ELLs 
meeting proficiency have negotiated 
with the US Department of Education to 
postpone reporting scores of newcomers. 

A causal inference would require, at a min- 
imum, that competing explanations be elim- 
inated. While the report notes these potential 
complications, regrettably these concerns do 
not prevent the authors from drawing specif- 
ic conclusions about the achievement trajec- 
tories of ELLs. 

VI. Review of the Validity of the 
Findings and Conclusions 

The conclusions about the academic perfor- 
mance of ELLs drawn in the report are not 
supported by the evidence presented for three 
specific reasons. The report underestimates 
the significance of limited English proficiency 
as a source of error in ELL test scores. Data 
are inappropriately presented as achievement 
trends for ELL students even though the com- 
position of student cohorts is different each 
year. Finally, the report’s authors reason that 
changes in percentages of students reaching 
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state benchmarks reflect actual changes in stu- 
dent achievement over time, even though 
competing explanations are not addressed. 

Given these limitations, it is inappropriate to 
draw conclusions of any kind from the data; 
therefore it is not only not useful, it is mis- 
leading to present summaries of the data, 
because readers will tend to draw conclu- 

o 

sions based on the data summaries alone. 



VII. Usefulness of the Report 
for Guidance of Policy 
and Practice 

The report is not a useful guide for policy or 
practice related to ELLs. The data analyses 
presented do not provide useful insight into 
ELL progress, and no judgments can be 
made regarding the relative success of cur- 
rent federal policy based on the report. 
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